{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## XGBoost算法速查表"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [1.读取libsvm格式数据并指定参数建模](#1.读取libsvm格式数据并指定参数建模)\n",
    "- [2.配合pandas的dataframe格式数据建模](#2.配合pandas的dataframe格式数据建模)\n",
    "- [3.使用XGBoost的SKLearn包](#3.使用XGBoost的SKLearn包)\n",
    "- [4.使用交叉验证查看模型训练效果](#4.使用交叉验证查看模型训练效果)\n",
    "- [5.添加预处理的交叉验证](#5.添加预处理的交叉验证)\n",
    "- [6.自定义损失函数与评估准则](#6.自定义损失函数与评估准则)\n",
    "- [7.使用前N颗树检测](#7.使用前N颗树检测)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.读取libsvm格式数据并指定参数建模\n",
    "libsvm用于保存稀疏矩阵格式的数据，数据格式如下：\n",
    "\n",
    "1 3:1 10:1\n",
    "\n",
    "第一列表示标签列，冒号前面的数字表示索引号，后面的数字表示具体的数值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[17:10:32] 6513x127 matrix with 143286 entries loaded from ./data/agaricus.txt.train\n",
      "[17:10:32] 1611x127 matrix with 35442 entries loaded from ./data/agaricus.txt.test\n"
     ]
    }
   ],
   "source": [
    "import xgboost as xgb\n",
    "# 将训练数据读取成矩阵形式\n",
    "data_train = xgb.DMatrix('./data/agaricus.txt.train')\n",
    "data_test = xgb.DMatrix('./data/agaricus.txt.test')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\teval-error:0.042831\ttrain-error:0.046522\n",
      "[1]\teval-error:0.021726\ttrain-error:0.022263\n",
      "[2]\teval-error:0.006207\ttrain-error:0.007063\n",
      "[3]\teval-error:0.018001\ttrain-error:0.0152\n"
     ]
    }
   ],
   "source": [
    "# booster算法的超参数\n",
    "param = {'max_depth':2, 'eta':1, 'objective':'binary:logistic'}\n",
    "num_boost_round = 4\n",
    "# 查看训练过程中的性能指标\n",
    "evals = [(data_test, 'eval'), (data_train, 'train')]\n",
    "# 模型训练\n",
    "bst = xgb.train(params=param, dtrain=data_train, num_boost_round=num_boost_round, evals=evals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "错误率为0.018001\n"
     ]
    }
   ],
   "source": [
    "# 使用模型预测\n",
    "preds = bst.predict(data_test)\n",
    "# 获取测试集的标签值\n",
    "labels = data_test.get_label()\n",
    "sum = 0\n",
    "for i in range(len(labels)):\n",
    "    if int(preds[i] > 0.5) != labels[i]:\n",
    "        sum += 1/float(len(preds))\n",
    "print('错误率为%f' % sum)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 模型保存\n",
    "bst.save_model('./model/001.model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.配合pandas的dataframe格式数据建模\n",
    "XGBoost需要使用矩阵格式的数据进行模型训练，下面是配合dataframe数据格式使用的样例做二分类，主要是将dataframe数据转为矩阵格式的数据\n",
    "- 数据如下：\n",
    "\n",
    "皮马印第安人糖尿病数据集 包含很多字段：怀孕次数 口服葡萄糖耐量试验中血浆葡萄糖浓度 舒张压（mm Hg） 三头肌组织褶厚度（mm）2小时血清胰岛素（μU/ ml） 体重指数（kg/（身高(m)^2） 糖尿病系统功能 年龄（岁）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>148</td>\n",
       "      <td>72</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>85</td>\n",
       "      <td>66</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>183</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>89</td>\n",
       "      <td>66</td>\n",
       "      <td>23</td>\n",
       "      <td>94</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>137</td>\n",
       "      <td>40</td>\n",
       "      <td>35</td>\n",
       "      <td>168</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Pregnancies  Glucose  BloodPressure  SkinThickness  Insulin   BMI  \\\n",
       "0            6      148             72             35        0  33.6   \n",
       "1            1       85             66             29        0  26.6   \n",
       "2            8      183             64              0        0  23.3   \n",
       "3            1       89             66             23       94  28.1   \n",
       "4            0      137             40             35      168  43.1   \n",
       "\n",
       "   DiabetesPedigreeFunction  Age  Outcome  \n",
       "0                     0.627   50        1  \n",
       "1                     0.351   31        0  \n",
       "2                     0.672   32        1  \n",
       "3                     0.167   21        0  \n",
       "4                     2.288   33        1  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "data = pd.read_csv('./data/Pima-Indians-Diabetes.csv')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "import xgboost as xgb\n",
    "# 切分训练集和测试集\n",
    "train, test = train_test_split(data)\n",
    "\n",
    "feature_col = ['Pregnancies', 'Glucose', 'BloodPressure', 'SkinThickness', 'Insulin', 'BMI', 'DiabetesPedigreeFunction', 'Age']\n",
    "target_col = ['Outcome']\n",
    "# dataframe 转 矩阵格式\n",
    "xg_train = xgb.DMatrix(train[feature_col].values, train[target_col].values)\n",
    "xg_test = xgb.DMatrix(test[feature_col].values, test[target_col].values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\teval-error:0.302083\ttrain-error:0.199653\n",
      "[1]\teval-error:0.28125\ttrain-error:0.166667\n",
      "[2]\teval-error:0.270833\ttrain-error:0.159722\n",
      "[3]\teval-error:0.229167\ttrain-error:0.163194\n"
     ]
    }
   ],
   "source": [
    "# booster超参数\n",
    "param = {'max_depth':5, 'eta':0.1, 'silent':1, 'subsample':0.7, 'colsample_bytree':0.7, 'objective':'binary:logistic'}\n",
    "# 训练轮次\n",
    "num_boost_round = 4\n",
    "# 查看训练过程中的性能指标\n",
    "evals = [(xg_test, 'eval'), (xg_train, 'train')]\n",
    "# 模型训练\n",
    "bst_02 = xgb.train(param, xg_train, num_boost_round, evals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "错误率为0.229167\n"
     ]
    }
   ],
   "source": [
    "# 模型预测\n",
    "preds = bst_02.predict(xg_test)\n",
    "# 判断准确率\n",
    "labels = xg_test.get_label()\n",
    "sum = 0\n",
    "for i in range(len(preds)):\n",
    "    if int(preds[i]>0.5) != labels[i]:\n",
    "        sum += 1 / float(len(labels))\n",
    "print('错误率为%f' % sum)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 模型保存\n",
    "bst_02.save_model('./model/002.model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.使用XGBoost的SKLearn包\n",
    "XGBoost直接train时需要使用矩阵格式的数据，当提前初始化创建xgboost模型时，可以直接使用array格式的数据进行训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import xgboost as xgb\n",
    "from sklearn.model_selection import train_test_split\n",
    "# Joblib是一组在Python中提供轻量级管道的工具\n",
    "import joblib as job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv('./data/Pima-Indians-Diabetes.csv')\n",
    "# 数据切分\n",
    "train, test = train_test_split(data)\n",
    "# 将dataframe格式的数据转为numpy格式的array格式\n",
    "feature_columns = ['Pregnancies', 'Glucose', 'BloodPressure', 'SkinThickness', 'Insulin', 'BMI', 'DiabetesPedigreeFunction', 'Age']\n",
    "target_column = 'Outcome'\n",
    "train_X = train[feature_columns].values\n",
    "train_y = train[target_column].values\n",
    "test_X = test[feature_columns].values\n",
    "test_y = test[target_column].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
       "              colsample_bynode=1, colsample_bytree=0.7, gamma=0,\n",
       "              learning_rate=0.1, max_delta_step=0, max_depth=4,\n",
       "              min_child_weight=1, missing=None, n_estimators=20, n_jobs=1,\n",
       "              nthread=None, objective='binary:logistic', random_state=0,\n",
       "              reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,\n",
       "              silent=None, subsample=0.7, verbosity=1)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 初始化模型\n",
    "xgb_classifier = xgb.XGBClassifier(n_estimators=20,\\\n",
    "                                   max_depth=4, \\\n",
    "                                   learning_rate=0.1, \\\n",
    "                                   subsample=0.7, \\\n",
    "                                   colsample_bytree=0.7)\n",
    "# 模型训练\n",
    "xgb_classifier.fit(train_X, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "错误类为0.239583\n"
     ]
    }
   ],
   "source": [
    "# 模型预测\n",
    "preds = xgb_classifier.predict(test_X)\n",
    "# 准确率计算\n",
    "print('错误类为%f' % ((preds!=test_y).sum()/float(test_y.shape[0])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['./model/003.model']"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 模型保存\n",
    "joblib.dump(xgb_classifier, './model/003.model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.使用交叉验证查看模型训练效果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>train-error-mean</th>\n",
       "      <th>train-error-std</th>\n",
       "      <th>train-rmse-mean</th>\n",
       "      <th>train-rmse-std</th>\n",
       "      <th>test-error-mean</th>\n",
       "      <th>test-error-std</th>\n",
       "      <th>test-rmse-mean</th>\n",
       "      <th>test-rmse-std</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.002572</td>\n",
       "      <td>0.000538</td>\n",
       "      <td>0.451012</td>\n",
       "      <td>0.000049</td>\n",
       "      <td>0.003224</td>\n",
       "      <td>0.001228</td>\n",
       "      <td>0.451068</td>\n",
       "      <td>0.000097</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.002687</td>\n",
       "      <td>0.001516</td>\n",
       "      <td>0.407984</td>\n",
       "      <td>0.000768</td>\n",
       "      <td>0.003992</td>\n",
       "      <td>0.002194</td>\n",
       "      <td>0.408091</td>\n",
       "      <td>0.000788</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.001228</td>\n",
       "      <td>0.000356</td>\n",
       "      <td>0.369007</td>\n",
       "      <td>0.000865</td>\n",
       "      <td>0.001995</td>\n",
       "      <td>0.001247</td>\n",
       "      <td>0.369173</td>\n",
       "      <td>0.000733</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.001113</td>\n",
       "      <td>0.000223</td>\n",
       "      <td>0.334152</td>\n",
       "      <td>0.000806</td>\n",
       "      <td>0.001842</td>\n",
       "      <td>0.000921</td>\n",
       "      <td>0.334506</td>\n",
       "      <td>0.000783</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.001267</td>\n",
       "      <td>0.000431</td>\n",
       "      <td>0.302530</td>\n",
       "      <td>0.000781</td>\n",
       "      <td>0.001381</td>\n",
       "      <td>0.000895</td>\n",
       "      <td>0.302882</td>\n",
       "      <td>0.000823</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>0.000921</td>\n",
       "      <td>0.000506</td>\n",
       "      <td>0.274560</td>\n",
       "      <td>0.001372</td>\n",
       "      <td>0.001228</td>\n",
       "      <td>0.001041</td>\n",
       "      <td>0.274913</td>\n",
       "      <td>0.001423</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.001152</td>\n",
       "      <td>0.000210</td>\n",
       "      <td>0.249184</td>\n",
       "      <td>0.001209</td>\n",
       "      <td>0.001228</td>\n",
       "      <td>0.001041</td>\n",
       "      <td>0.249552</td>\n",
       "      <td>0.001270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.001152</td>\n",
       "      <td>0.000210</td>\n",
       "      <td>0.226123</td>\n",
       "      <td>0.001325</td>\n",
       "      <td>0.001228</td>\n",
       "      <td>0.001041</td>\n",
       "      <td>0.226577</td>\n",
       "      <td>0.001178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.001152</td>\n",
       "      <td>0.000210</td>\n",
       "      <td>0.205030</td>\n",
       "      <td>0.001077</td>\n",
       "      <td>0.001228</td>\n",
       "      <td>0.001041</td>\n",
       "      <td>0.205654</td>\n",
       "      <td>0.001250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.001152</td>\n",
       "      <td>0.000210</td>\n",
       "      <td>0.186308</td>\n",
       "      <td>0.000957</td>\n",
       "      <td>0.001228</td>\n",
       "      <td>0.001041</td>\n",
       "      <td>0.187019</td>\n",
       "      <td>0.001120</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   train-error-mean  train-error-std  train-rmse-mean  train-rmse-std  \\\n",
       "0          0.002572         0.000538         0.451012        0.000049   \n",
       "1          0.002687         0.001516         0.407984        0.000768   \n",
       "2          0.001228         0.000356         0.369007        0.000865   \n",
       "3          0.001113         0.000223         0.334152        0.000806   \n",
       "4          0.001267         0.000431         0.302530        0.000781   \n",
       "5          0.000921         0.000506         0.274560        0.001372   \n",
       "6          0.001152         0.000210         0.249184        0.001209   \n",
       "7          0.001152         0.000210         0.226123        0.001325   \n",
       "8          0.001152         0.000210         0.205030        0.001077   \n",
       "9          0.001152         0.000210         0.186308        0.000957   \n",
       "\n",
       "   test-error-mean  test-error-std  test-rmse-mean  test-rmse-std  \n",
       "0         0.003224        0.001228        0.451068       0.000097  \n",
       "1         0.003992        0.002194        0.408091       0.000788  \n",
       "2         0.001995        0.001247        0.369173       0.000733  \n",
       "3         0.001842        0.000921        0.334506       0.000783  \n",
       "4         0.001381        0.000895        0.302882       0.000823  \n",
       "5         0.001228        0.001041        0.274913       0.001423  \n",
       "6         0.001228        0.001041        0.249552       0.001270  \n",
       "7         0.001228        0.001041        0.226577       0.001178  \n",
       "8         0.001228        0.001041        0.205654       0.001250  \n",
       "9         0.001228        0.001041        0.187019       0.001120  "
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb.cv(param, data_train, num_boost_round=10, nfold=5, metrics={'error', 'rmse'}, seed=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.添加预处理的交叉验证"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function cv in module xgboost.training:\n",
      "\n",
      "cv(params, dtrain, num_boost_round=10, nfold=3, stratified=False, folds=None, metrics=(), obj=None, feval=None, maximize=False, early_stopping_rounds=None, fpreproc=None, as_pandas=True, verbose_eval=None, show_stdv=True, seed=0, callbacks=None, shuffle=True)\n",
      "    Cross-validation with given parameters.\n",
      "    \n",
      "    Parameters\n",
      "    ----------\n",
      "    params : dict\n",
      "        Booster params.\n",
      "    dtrain : DMatrix\n",
      "        Data to be trained.\n",
      "    num_boost_round : int\n",
      "        Number of boosting iterations.\n",
      "    nfold : int\n",
      "        Number of folds in CV.\n",
      "    stratified : bool\n",
      "        Perform stratified sampling.\n",
      "    folds : a KFold or StratifiedKFold instance or list of fold indices\n",
      "        Sklearn KFolds or StratifiedKFolds object.\n",
      "        Alternatively may explicitly pass sample indices for each fold.\n",
      "        For ``n`` folds, **folds** should be a length ``n`` list of tuples.\n",
      "        Each tuple is ``(in,out)`` where ``in`` is a list of indices to be used\n",
      "        as the training samples for the ``n`` th fold and ``out`` is a list of\n",
      "        indices to be used as the testing samples for the ``n`` th fold.\n",
      "    metrics : string or list of strings\n",
      "        Evaluation metrics to be watched in CV.\n",
      "    obj : function\n",
      "        Custom objective function.\n",
      "    feval : function\n",
      "        Custom evaluation function.\n",
      "    maximize : bool\n",
      "        Whether to maximize feval.\n",
      "    early_stopping_rounds: int\n",
      "        Activates early stopping. CV error needs to decrease at least\n",
      "        every <early_stopping_rounds> round(s) to continue.\n",
      "        Last entry in evaluation history is the one from best iteration.\n",
      "    fpreproc : function\n",
      "        Preprocessing function that takes (dtrain, dtest, param) and returns\n",
      "        transformed versions of those.\n",
      "    as_pandas : bool, default True\n",
      "        Return pd.DataFrame when pandas is installed.\n",
      "        If False or pandas is not installed, return np.ndarray\n",
      "    verbose_eval : bool, int, or None, default None\n",
      "        Whether to display the progress. If None, progress will be displayed\n",
      "        when np.ndarray is returned. If True, progress will be displayed at\n",
      "        boosting stage. If an integer is given, progress will be displayed\n",
      "        at every given `verbose_eval` boosting stage.\n",
      "    show_stdv : bool, default True\n",
      "        Whether to display the standard deviation in progress.\n",
      "        Results are not affected, and always contains std.\n",
      "    seed : int\n",
      "        Seed used to generate the folds (passed to numpy.random.seed).\n",
      "    callbacks : list of callback functions\n",
      "        List of callback functions that are applied at end of each iteration.\n",
      "        It is possible to use predefined callbacks by using\n",
      "        :ref:`Callback API <callback_api>`.\n",
      "        Example:\n",
      "    \n",
      "        .. code-block:: python\n",
      "    \n",
      "            [xgb.callback.reset_learning_rate(custom_rates)]\n",
      "    shuffle : bool\n",
      "        Shuffle data before creating folds.\n",
      "    \n",
      "    Returns\n",
      "    -------\n",
      "    evaluation history : list(string)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(xgb.cv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>train-auc-mean</th>\n",
       "      <th>train-auc-std</th>\n",
       "      <th>test-auc-mean</th>\n",
       "      <th>test-auc-std</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.999928</td>\n",
       "      <td>0.000030</td>\n",
       "      <td>0.999917</td>\n",
       "      <td>0.000035</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.999986</td>\n",
       "      <td>0.000012</td>\n",
       "      <td>0.999977</td>\n",
       "      <td>0.000015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.999976</td>\n",
       "      <td>0.000007</td>\n",
       "      <td>0.999965</td>\n",
       "      <td>0.000019</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.999994</td>\n",
       "      <td>0.000012</td>\n",
       "      <td>0.999977</td>\n",
       "      <td>0.000042</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.999993</td>\n",
       "      <td>0.000015</td>\n",
       "      <td>0.999975</td>\n",
       "      <td>0.000051</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>0.999993</td>\n",
       "      <td>0.000015</td>\n",
       "      <td>0.999975</td>\n",
       "      <td>0.000051</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.999993</td>\n",
       "      <td>0.000015</td>\n",
       "      <td>0.999975</td>\n",
       "      <td>0.000051</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.999993</td>\n",
       "      <td>0.000015</td>\n",
       "      <td>0.999975</td>\n",
       "      <td>0.000051</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.999994</td>\n",
       "      <td>0.000011</td>\n",
       "      <td>0.999969</td>\n",
       "      <td>0.000062</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.999994</td>\n",
       "      <td>0.000012</td>\n",
       "      <td>0.999969</td>\n",
       "      <td>0.000062</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   train-auc-mean  train-auc-std  test-auc-mean  test-auc-std\n",
       "0        0.999928       0.000030       0.999917      0.000035\n",
       "1        0.999986       0.000012       0.999977      0.000015\n",
       "2        0.999976       0.000007       0.999965      0.000019\n",
       "3        0.999994       0.000012       0.999977      0.000042\n",
       "4        0.999993       0.000015       0.999975      0.000051\n",
       "5        0.999993       0.000015       0.999975      0.000051\n",
       "6        0.999993       0.000015       0.999975      0.000051\n",
       "7        0.999993       0.000015       0.999975      0.000051\n",
       "8        0.999994       0.000011       0.999969      0.000062\n",
       "9        0.999994       0.000012       0.999969      0.000062"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "# 自定义预处理函数：计算正负样本权重比\n",
    "def fpreproc(dtrain, dtest, param):\n",
    "    label = dtrain.get_label()\n",
    "    ratio = float(np.sum(label == 0)) / np.sum(label == 1)\n",
    "    param['scala_pos_weight'] = ratio\n",
    "    return (dtrain, dtest, param)\n",
    "# 先做预处理，计算样本权重，在做交叉验证\n",
    "xgb.cv(param, data_train, num_boost_round=10, nfold=5, metrics={'auc'}, seed=0, fpreproc=fpreproc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.自定义损失函数与评估准则"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "running cross validation, with cutomsized loss function\n",
      "[0]\teval-rmse:0.306902\ttrain-rmse:0.306163\teval-error:0.518312\ttrain-error:0.517887\n",
      "[1]\teval-rmse:0.17919\ttrain-rmse:0.177276\teval-error:0.518312\ttrain-error:0.517887\n",
      "[2]\teval-rmse:0.172566\ttrain-rmse:0.171727\teval-error:0.016139\ttrain-error:0.014433\n",
      "[3]\teval-rmse:0.269611\ttrain-rmse:0.271113\teval-error:0.016139\ttrain-error:0.014433\n",
      "[4]\teval-rmse:0.396904\ttrain-rmse:0.398245\teval-error:0.016139\ttrain-error:0.014433\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>train-error-mean</th>\n",
       "      <th>train-error-std</th>\n",
       "      <th>train-rmse-mean</th>\n",
       "      <th>train-rmse-std</th>\n",
       "      <th>test-error-mean</th>\n",
       "      <th>test-error-std</th>\n",
       "      <th>test-rmse-mean</th>\n",
       "      <th>test-rmse-std</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.517887</td>\n",
       "      <td>0.001085</td>\n",
       "      <td>0.308880</td>\n",
       "      <td>0.005170</td>\n",
       "      <td>0.517886</td>\n",
       "      <td>0.004343</td>\n",
       "      <td>0.309038</td>\n",
       "      <td>0.005207</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.517887</td>\n",
       "      <td>0.001085</td>\n",
       "      <td>0.176504</td>\n",
       "      <td>0.002046</td>\n",
       "      <td>0.517886</td>\n",
       "      <td>0.004343</td>\n",
       "      <td>0.177802</td>\n",
       "      <td>0.003767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.014433</td>\n",
       "      <td>0.000223</td>\n",
       "      <td>0.172680</td>\n",
       "      <td>0.003719</td>\n",
       "      <td>0.014433</td>\n",
       "      <td>0.000892</td>\n",
       "      <td>0.174890</td>\n",
       "      <td>0.009391</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.014433</td>\n",
       "      <td>0.000223</td>\n",
       "      <td>0.275761</td>\n",
       "      <td>0.001776</td>\n",
       "      <td>0.014433</td>\n",
       "      <td>0.000892</td>\n",
       "      <td>0.276689</td>\n",
       "      <td>0.005918</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.014433</td>\n",
       "      <td>0.000223</td>\n",
       "      <td>0.399889</td>\n",
       "      <td>0.003369</td>\n",
       "      <td>0.014433</td>\n",
       "      <td>0.000892</td>\n",
       "      <td>0.400118</td>\n",
       "      <td>0.006243</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   train-error-mean  train-error-std  train-rmse-mean  train-rmse-std  \\\n",
       "0          0.517887         0.001085         0.308880        0.005170   \n",
       "1          0.517887         0.001085         0.176504        0.002046   \n",
       "2          0.014433         0.000223         0.172680        0.003719   \n",
       "3          0.014433         0.000223         0.275761        0.001776   \n",
       "4          0.014433         0.000223         0.399889        0.003369   \n",
       "\n",
       "   test-error-mean  test-error-std  test-rmse-mean  test-rmse-std  \n",
       "0         0.517886        0.004343        0.309038       0.005207  \n",
       "1         0.517886        0.004343        0.177802       0.003767  \n",
       "2         0.014433        0.000892        0.174890       0.009391  \n",
       "3         0.014433        0.000892        0.276689       0.005918  \n",
       "4         0.014433        0.000892        0.400118       0.006243  "
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print ('running cross validation, with cutomsized loss function')\n",
    "# 自定义损失函数，需要提供损失函数的一阶导和二阶导\n",
    "def logregobj(preds, dtrain):\n",
    "    labels = dtrain.get_label()\n",
    "    preds = 1.0 / (1.0 + np.exp(-preds))\n",
    "    grad = preds - labels\n",
    "    hess = preds * (1.0-preds)\n",
    "    return grad, hess\n",
    "\n",
    "# 自定义评估准则，评估预估值和标准答案之间的差距\n",
    "def evalerror(preds, dtrain):\n",
    "    labels = dtrain.get_label()\n",
    "    sum = 0\n",
    "    for i in range(len(preds)):\n",
    "        if labels[i] != (preds[i] > 0):\n",
    "            sum += 1 / float(len(labels))\n",
    "    return 'error', sum\n",
    "\n",
    "evals  = [(data_test,'eval'), (data_train,'train')]\n",
    "param = {'max_depth':3, 'eta':0.1, 'silent':1}\n",
    "num_boost_round = 5\n",
    "# 自定义损失函数训练\n",
    "bst = xgb.train(param, data_train, num_boost_round, evals, obj=logregobj, feval=evalerror)\n",
    "# 交叉验证\n",
    "xgb.cv(param, data_train, num_boost_round, nfold = 5, seed = 0,\n",
    "       obj = logregobj, feval=evalerror)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.使用前N颗树检测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\teval-error:0.260417\ttrain-error:0.223958\n",
      "[1]\teval-error:0.260417\ttrain-error:0.180556\n",
      "[2]\teval-error:0.208333\ttrain-error:0.161458\n",
      "[3]\teval-error:0.203125\ttrain-error:0.152778\n",
      "[4]\teval-error:0.203125\ttrain-error:0.137153\n",
      "[5]\teval-error:0.213542\ttrain-error:0.131944\n",
      "[6]\teval-error:0.213542\ttrain-error:0.133681\n",
      "[7]\teval-error:0.213542\ttrain-error:0.131944\n",
      "[8]\teval-error:0.203125\ttrain-error:0.137153\n",
      "[9]\teval-error:0.192708\ttrain-error:0.144097\n",
      "用前1颗树预测的错误率为 0.260417\n",
      "用前9颗树预测的错误率为 0.203125\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pickle\n",
    "import xgboost as xgb\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# 基本例子，从csv文件中读取数据，做二分类\n",
    "\n",
    "# 用pandas读入数据\n",
    "data = pd.read_csv('./data/Pima-Indians-Diabetes.csv')\n",
    "\n",
    "# 做数据切分\n",
    "train, test = train_test_split(data)\n",
    "\n",
    "# 转换成Dmatrix格式\n",
    "feature_columns = ['Pregnancies', 'Glucose', 'BloodPressure', 'SkinThickness', 'Insulin', 'BMI', 'DiabetesPedigreeFunction', 'Age']\n",
    "target_column = 'Outcome'\n",
    "xgtrain = xgb.DMatrix(train[feature_columns].values, train[target_column].values)\n",
    "xgtest = xgb.DMatrix(test[feature_columns].values, test[target_column].values)\n",
    "\n",
    "#参数设定\n",
    "param = {'max_depth':5, 'eta':0.1, 'silent':1, 'subsample':0.7, 'colsample_bytree':0.7, 'objective':'binary:logistic' }\n",
    "\n",
    "# 设定watchlist用于查看模型状态\n",
    "evals  = [(xgtest,'eval'), (xgtrain,'train')]\n",
    "num_boost_round = 10\n",
    "bst = xgb.train(param, xgtrain, num_boost_round, evals)\n",
    "\n",
    "# 只用第1颗树预测\n",
    "ypred1 = bst.predict(xgtest, ntree_limit=1)\n",
    "# 用前9颗树预测\n",
    "ypred2 = bst.predict(xgtest, ntree_limit=9)\n",
    "label = xgtest.get_label()\n",
    "print ('用前1颗树预测的错误率为 %f' % (np.sum((ypred1>0.5)!=label) /float(len(label))))\n",
    "print ('用前9颗树预测的错误率为 %f' % (np.sum((ypred2>0.5)!=label) /float(len(label))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
